Neural networks are electronic circuits for processing information and emulate the functions of the biological neuron of the human brain. They provide an approach which is an alternative to information processing in the modern and popular digital computers in which information is processed by the step by step execution of a series of programmed instructions. Neural networks possess large computational power which is more suitable for some information processing applications, such as image, speech, and pattern recognition, communication systems, machine vision, and robotics.
In a neural network many individual computing elements called neurons are densely interconnected. Typically, each neuron receives several inputs from many other neurons and has a single output, although the output may be both an output voltage and its complement. In a generalized neural network, each neuron receives an input from the output of each other neuron. Each coupling from a neuron output to a neuron input may be strong or weak and its strength, called the synaptic weight, is desirably controllable over a range of values.
Hopfield proposed a neural network circuit model in, e.g. "Simple Neural Optimization Networks: an A/D converter, a signal decision circuit, and a linear programming circuit" by J. J. Hopfield and D. W. Tank, IEEE Transactions on Circuits and Systems, vol. CAS-33, No. 5, May 1986, pages 533-541. FIG. 7 illustrates an application of the present invention to a neural network of the Hopfield type and therefore may be referred to in connection with an explanation of the Hopfield neural network model.
FIG. 7 illustrates an n-neuron neural network, each neuron including a multiplier/summer, boxed in dashed lines, the output of which is connected to a double inverter. For example, neuron 1 has a multiplier/summer 10 and a double inverter 12. Also illustrated are a parallel resistance and capacitance 14 representing the parasitic input impedance inherent in a practically implemented double inverter.
The synaptic weight Y.sub.ij defines for each neuron input the extent to which an input of the i-th neuron is driven by the output of the j-th neuron. Thus, for
example, in FIG. 7 Y.sub.11 is the extent to which the input to neuron 1 from the output of neuron 1 drives the first neuron. A negative value of Y.sub.ij indicates that the j-th neuron inhibits the i-th neuron. Thus, each input to a neuron is represented by the product of two multiplicand voltages, one multiplicand being an input from a neuron circuit output and the other multiplicand being the synaptic weight assigned to that input. The synaptic weight is controlled off chip by the application of a DC control voltage of a suitable level for selecting the desired synaptic weight. The multiplier/summer circuit component of each neuron circuit, i.e. the i-th neuron, desirably achieves a vector scalar product in the form: ##EQU1## where X.sub.j is the output from the j-th neuron and Y.sub.ij is the assigned positive or negative synaptic weight that is realized through voltage levels.
The output of the multiplier/summer represents the product of two n-tuple vector inputs. One n-tuple vector input is represented by its vector components Y.sub.11, Y.sub.12, . . . Y.sub.1n, and the other vector input is represented by its vector components X.sub.1, X.sub.2, and X.sub.n. Thus, desirably each multiplier/summer is an analog device, the output of which provides the sum of products represented by equation I. This product is in the form of the well known vector or cross product and scalar or dot product used in vector analysis.
As is known in the prior art, a double inverter, such as double inverter 12 in FIG. 7, may be utilized in the realization of the neuron circuit. The double inverter provides an output which is a sigmoidal function of its input. It is a very high gain amplifier providing an S-shaped non-linearity. It operates as a non-linear limiter to provide an output at one of two discrete output levels, +V or -V, depending upon whether its input is greater than or less than a selected level.
A neuron network of the Hopfield type will settle into one of a limited number of steady-state equilibria for each set of inputs and synaptic weights. The synaptic weights are desirably programmable off chip by application of suitable voltage levels to enable the neural network to be designed for the solution of various problems. Changing the synaptic weight alters the location and number of the steady-state equilibria of the network.
Because neural networks are highly complex and have dense interconnectivities, the only practical implementation is in very large scale integrated circuits. It is therefore highly desirable that such circuits be implemented in metal oxide semiconductor, field effect transistor form, utilizing the fewest possible number of FET devices, while preserving the basic functional properties of the neuron circuit. It is also desirable that they use simple analog component cells that make effective use of CAD tools, occupy small area and are versatile, programmable, easy to design, and minimize the interconnectivities between the cells.
In the patent application of Salam and Ismail, Ser. No. 348,309, filed May 5, 1989, a multiplier circuit and feedback neural networks using their circuit were disclosed. However, the circuit of that patent application requires the use of twice as many FET devices as are utilized in the circuit of the present invention.